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Abstract.  We  introduce  a  novel  representation  of  two-dimensional  shape  that  we 
call  smoothed  local  symmetries  (SLS).  Smoothed  local  symmetries  represent  both 
the  bounding  contour  of  a  shape  fragment  and  the  region  that  it  occupies.  In 
this  paper  we  develop  the  main  features  of  the  SLS  representation  and  describe 
an  implemented  algorithm  that  computes  it.  The  performance  of  the  algorithm  is 
illustrated  for  a  set  of  tools.  We  conclude  by  sketching  a  method  for  determining 
the  articulation  of  a  shape  into  subshapes. 
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1.  Introduction 


We  introduce  a  novel  representation  of  two-diinensional  shape  that  we  call  s»  wothed 
local  symmetries  (SLS).  Smoothed  local  symmetries  represent  both  the  hi  unding 
contour  of  a  shape  fragment  and  the  region  that  it  subtends  or  cm  lo-a-s  In  this 
paper  we  develop  the  main  features  of  the  SLS  representation  and  describe  an 
implemented  algorithm  that  computes  it.  The  performance  ol  the  algorithm  is 
illustrated  for  a  set  of  tools.  One  step  in  the  algorithm  is  the  cons' r  ution  of  a 
representation  of  the  significant  changes  in  curvature  along  the  contour  of  the 
shape.  We  call  that  representation  the  curvature  primal  sketch  [Asada  am  Brady 
1984],  since  it  is  analogous  to  the  primal  sketch  representation  of  intensity  hanges 
advocated  by  Marr  [1976].  We  conclude  by  sketching  a  method  for  detcro doing  the 
articulation  of  a  shape  into  subshapes.  A  preliminary  version  of  this  p.-pm  appeared 
as  [Brady  and  Asada,  1984]. 

The  representation  of  shape  is  a  fundamental  problem  in  computer  \isiun  that 
has  been  surprisingly  neglected.  It  is  becoming  increasingly  important  ir.  Robot 
vision  as  the  inadequacies  of  currently  available  systems  manifest  themselves.  So 
long  as  the  position  and  orientation  of  objects  is  tightly  constrained,  so  long  as 
contrast  is  sufficient  to  allow  trivial  segmentation  into  figure  and  ground,  and  so 
long  as  objects  do  not  overlap,  simple  vision  systems  suffice.  Increasing  uncertainty 
in  any  of  these  variables  requires  a  substantial  increase  in  system  sophistication. 

Fortunately,  there  has  been  considerable  advance  in  our  understanding  of 
computer  vision  over  the  past  decade  (see  [Ballard  and  Brown  1982,  Brady  1982a, 
Marr  1982]  for  reviews).  For  example,  edge  finding,  though  far  from  being  a  solved 
problem,  is  increasingly  reliable  and  able  to  support  the  accurate  computation  of 
curvature  (see  below).  Better  edge  finders  are  a  prerequisite  to  improved  shape 
representations.  The  additional  information  that  they  provide  supports  richer 
representations  of  shape,  such  as  that  discussed  here.  Much  attention  has  focussed 
on  three-dimensional  vision.  There  has  been  significant  progress  in  computing 
depth  (or  local  surface  orientation)  using  stereo,  photometric  stereo,  structured 
light,  and  bounding  contour.  In  other  work,  we  build  on  these  additional  capabilities 
of  computer  vision  to  develop  better  representations  of  three  dimensional  shape 
[Brady  and  Yuille  1984]. 

A  representation  of  shape  can  be  judged  according  to  its  suitability  for  many 
uses  in  Robotics: 

•  Recognition.  Given  a  database  of  models,  recognition  amounts  to  associating 
an  object  with  its  model.  There  are  two  principal  difficulties.  First,  objects  may 
overlap.  An  object  has  to  be  recognised  from  its  non-overlapped  portions  and 
these  portions  cannot  be  predicted  in  advance.  The  shape  of  each  portion  needs 
to  be  represented,  and  provides  constraint  on  model  identification.  The  richer  the 
representation  of  a  visible  fragment,  the  tighter  the  constraint.  Recognition  involves 
the  summation  of  the  constraints’  from  the  separate  visible  portions.  It  is  best 
suited  to  a  highly  parallel  computer  such  as  the  connection  machine  [Hillis  1981]. 
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Representing  a  shape  by  a  set  of  global  features  such  as  its  moments  cannot  deal 
with  overlap. 

Second,  in  general,  the  database  of  models  must  allow  for  variations  in  shape. 
For  example,  not  all  tack  hammers  are  exactly  alike.  Also,  tack  hammers  are 
clearly  distinguishable  from  Warrington  hammers  and  ball  poen  hammers,  but  they 
are  equally  all  clearly  identifiable  as  hammers.  Artificial  Intelligence  has  developed 
semantic  networks  to  represent  classes  and  subclasses  of  objects  ;  nd  their  subparts. 
In  particular,  Brooks  and  Binford  [1980]  developed  ACRONYM  to  address  such 
problems.  This  issue  will  be  taken  up  again  in  Section  6. 

•  Inspection.  Since  a  part  to  be  inspected  may  have  defects,  it  is  unreasonable  to 
expect  a  perfect  match  between  a  shape  description  and  its  model.  We  need  to  be 
able  to  recognize  the  model  corresponding  to  a  flawed  object  and  to  determine  the 
nature  of  the  flaw.  The  richer  the  representation  of  shape  that  can  be  computed  for 
an  object,  the  greater  the  possibility  of  automatically  generating  an  explanation  of 
the  flaw  from  the  mismatch  between  the  object  description  and  its  model.  This  has 
been  a  recurring  theme  in  Artificial  Intelligence  [Winston  1984].  Flaws  are  typically 
restricted  to  a  local  portion  of  a  shape.  The  shape  representation  needs  to  be  stable 
with  respect  to  local  changes,  yet  sensitive  enough  to  represent  them  [Marr  and 
Nishihara  1978].  We  say  that  the  shape  representation  must  have  local  support. 

•  Grasping.  Part  of  our  understanding  of  a  tool  such  as  a  screwdriver  is  how 
to  pick  it  up,  and  how  to  grasp  it  when  we  use  it.  For  a  given  end-effector,  one 
can  imagine  motor  program  links  in  the  model  of  an  object  that  contain  such 
information.  However,  faced  with  an  object  that  we  have  never  seen  before,  we  still 
grasp  it  sensibly.  In  order  to  do  this,  we  must  determine  a  grasp  position  from  a 
representation  of  the  shape  of  the  object  to  be  grasped.  Boissonat  [1982]  and  Brady 
[1982b]  have  suggested  ways  in  which  this  might  be  done. 

•  Reasoning.  Robotics  is  the  intelligent  connection  of  perception  to  action. 
Intelligent  robots  need  to  be  able  to  reason  about  the  tasks  that  they  are  to 
perform,  for  example  to  plan  processing  and  assembly  sequences.  Currently  popular 
models  of  reasoning,  such  as  expert  systems,  are  typically  supplied  with  pre-digested 
sensory  information.  Shape  representations  should  support  enhanced  reasoning  (see 
for  example  [Brady  1984b]). 

In  previous  papers  [Brady  1982b,  1982c,  1983]  we  have  introduced  a 
representation  of  two-dimensional  shape  that  we  call  smoothed  local  symmetries 
(SLS).  Section  2  summarizes  the  main  ideas,  and  then  develops  a  parametric  analysis 
of  the  SLS  of  a  shape.  The  parametric  description  is  both  region  and  contour  based. 
In  Section  3,  we  describe  an  implementation  of  the  representation,  illustrating  the 
performance  of  our  programs  in  Section  5  using  a  set  of  hand  tools.  We  discuss  in 
Section  4  the  reasons  for  computing  a  representation,  called  the  curvature  primal 
sketch,  of  the  significant  curvature  changes  along  a  contour.  Asada  and  Brady 
[1984]  discuss  the  curvature  primal  sketch  in  more  detail.  Finally,  in  Section  6  we 
sketch  a  method,  currently  being  implemented,  for  determining  the  articulation  of 
a  shape  into  subshapes.  The  emphasis  of  this  paper  is  on  competence:  we  develop 
an  implemented  representation  that  can  support  a  variety  of  applications  without 
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excessive  concern  for  ’’real  time”  performance.  Hardware  implementation  of  the 
ideas  developed  here  will  be  discussed  elsewhere. 


2.  Smoothed  local  symmetries 


2.1.  Background  on  shape  description 

Shape  is  represented  in  most  current  robotics  or  commercially  available  vision 
systems  [Agin  1980,  Holland,  Rossol,  and  Ward  1979,  Perkins  1978,  Rosenfold  1982] 
in  terms  of  global  features  such  as:  the  center  of  area,  number  of  holes,  tlu  aspect 
ratio  of  the  principal  axes,  and  the  ratio  of  the  perimeter  squared  to  ; he  area 
Such  features  can  be  computed  efficiently  and  they  are  reasonably  insensitive  to 
noise  and  quantization,  even  for  low  resolution  binary  images.  Unfortunately,  the 
computed  value  of  a  global  feature  for  the  visible  portion  of  an  occluded  object 
bears  an  arbitrary  relationship  to  the  value  that  would  be  computed  for  the  whole 
object.  It  follows  that  it  is  at  best  difficult,  and  usually  impossible,  to  recognize 
occluded  parts  using  global  features. 

Bolles  and  Cain  [1982]  uses  “focus  features”  to  handle  limited  cases  of  occlusion. 
An  object  model  is  a  graph  whose  nodes  are  features  such  as  corners  and  holes  and 
whose  edges  are  the  exact  distance  and  relative  orientation  of  the  two  nodes  related 
by  the  edge.  Recognition  is  implemented  by  finding  maximal  cliques  in  model 
graphs.  In  essence,  Bolles’  representation  is  a  compromise  between  global  features 
and  representations  that  have  local  support,  such  as  smoothed  local  symmetries. 
The  representation  is,  however,  metrically  exact  and  uses  only  simple  features  such 
as  rectangular  corners,  holes,  and  their  relative  position.  It  is  restricted  to  straight 
line  contours. 

Nearly  every  representation  of  shape  can  be  classified  as  being  either  contour 
based  or  region  based.  Contour  and  region  based  representations  of  shape  make 
different  aspects  of  shape  explicit.  Contour  based  representations  encode  the 
bounding  contour  of  a  shape.  Examples  include  Fourier  series  expansions  of  the 
contour,  chain  encoding,  and  spline  approximations  (see  Pavlidis  [1978]).  Asada  and 
Brady  [1984]  have  developed  a  representation  that  makes  explicit  the  significant 
curvature  changes  along  a  contour.  Attneave  [1954]  showed  that  such  changes, 
essentially  the  curvature  analogue  of  the  primal  sketch  (see  [Asada  and  Brady 
1984]),  carry  a  great  deal  of  information  for  the  human  visual  system.  Certainly, 
the  human  visual  system  is  remarkably  sensitive  to  curvature  changes  [Watt  and 
Andrews  1982].  It  is  difficult,  however,  to  compute  or  determine  important  regional 
properties,  such  as  symmetry,  or  descriptions  such  as  “elongated  and  curved”,  in 
a  purely  contour  based  representation  of  shape. 

Region  based  approaches  represent  a  shape  by  encoding  the  two-dimensional 
space  occupied  by  the  shape.  Examples  include  quadtrees  [Rosenfeld  and  Kak  1982], 
the  symmetric  axis  transform  (SAT)  [Blum  and  Nagel  1978],  and  (two-dimensional) 
generalized  cones  [Nevatiaand  Binford  1977,  Brooks  1981].  The  SAT  and  generalized 
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Figure  1.  a.  A  hammer  claw.  b.  Its  geometric  representation.  The  solid  lines  amount  to  a  contour 
baaed  representation,  while  the  dotted  lines  encode  the  region  occupied  by  the  claw. 

cylinders  emphasize  elongation,  describing  a  shape  by  a  one-dimensional  axis  and 
the  disposition  of  the  shape  about  that  axis.  Occlusion  poses  a  difficult  problem 
for  purely  region  based  representations,  since  recognizing  partly  occluded  shapes 
requires  a  representation  that  has  local  support  and  a  parametric  description  that 
is  in  part  contour  based.  The  smoothed  local  symmetries  representation,  described 
\n  the  next  section,  is  both  contour  based  and  region  based. 

2.2.  Definition  of  smoothed  local  symmetries 

Figure  1  shows  a  hammer  claw  and  its  geometric  representation.  The  arc  AB  is 
described  as  a  convex  arc  whose  curvature  is  constant.  In  our  implementations,  arcs 
are  described  either  by  circles  or  by  Cornu  spirals,  for  which  curvature  varies  linearly 
with  arc  length.  The  total  angular  change  along  AB  is  recorded.  The  discontinuity 
in  curvature  at  B  is  made  explicit.  There  are  various  kinds  of  primitive  curvature 
discontinuities:  in  this  case,  B  is  an  acute  angled  corner.  Other  primitive  curvature 
discontinuities  are  defined  below.  They  include  ends,  cranks,  smooth  joins,  bumps, 
and  dents.  The  arc  BC  is  concave,  and  is  also  described  by  the  best  fitting 
circle.  These  descriptors  form  a  contour  based  representation.  The  region  based 
description  is  shown  between  the  representations  of  AB  and  BC.  It  is  described  as 
a  smoothed  local  symmetry  of  primitive  type  beak.  Other  primitive  region  types 
are  defined  below.  They  include  cup,  sector,  wedge,  plinth,  and  flare.  The  region 
description  has  several  parameters  associated  with  it,  including  a  measure  of  the 
width  of  the  region,  and  the  curvature  of  an  associated  axis.  In  this  case,  we  note 
that  the  width  decreases  monotonically  in  the  direction  approaching  B,  and  has  an 
axis  that  “slowly”  curves  to  the  right.  Curvature  is  quantized  to  5  levels,  following 
Hollerbach  [1975].  For  a  fuller  account  of  parametric  descriptions,  see  Heide  [1984]. 

The  definition  of  smoothed  local  symmetries  is  in  three  parts.  First,  Figure 
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2a  shows  the  geometry  of  a  local  symmetry.  A  and  B  are  points  on  the  bounding 
contour  of  a  shape.  The  angle  a  between  the  line  BA  and  the  outward  normal 
at  A  is  the  same  as  the  angle  between  BA  and  the  inward  normal  n#  at  B.  (A 
precise  definition  is  given  in  the  next  section.)  Figure  2b  illustrates  that  in  general, 
there  may  be  several  points  Bt  forming  local  symmetries  with  a  given  point  A. 

Implicit  in  the  definition  of  local  symmetry  is  the  requirement  that  the  bounding 
contour  of  the  shape  be  extracted  and  the  tangent  angle  computed  sufficiently 
accurately.  We  use  the  edge  finder  developed  recently  by  Canny  [1983].  This 
extremizes  the  product  of  the  signal-to-noise  response  at  a  step  edge  and  a  measure 
of  the  localization  of  the  edge,  while  producing  a  single  response  to  a  single  step. 
Earlier  work  used  a  non-directional  operator  based  on  a  difference  of  Gaussians 
filter,  proposed  by  Marr  and  Hildreth  [1980].  The  Canny  operator  is  preferred 
because  it  gives  more  accurate  tangent  directions  and  is  less  susceptible  to  noise. 
For  more  details,  see  [Canny  1983]. 

Second,  we  consider  the  loci  of  local  symmetries  that  are  maximal  with  respect 
to  forming  a  smooth  curve.  We  call  such  loci  axes  or  spines.  Figure  3a  shows  the 
axis  for  a  portion  of  a  contour  that  contains  a  corner.  Each  axis  is  an  alternative, 
locally  plausible  way  to  describe  some  piece  of  the  contour  and  the  region  subtended 
by  the  axis.  We  call  this  portion  of  the  shape,  consisting  of  a  part  of  the  contour 
and  a  subregion  of  the  shape,  the  cover  of  the  axis.  Notice  that  the  covers  of  certain 
axes  are  wholly,  that  is,  properly,  contained  in  the  covers  of  others.  In  such  cases, 
we  say  that  the  axis  with  the  contained  cover  is  subsumed  by  the  axis  with  the 
containing  cover.  Figure  3b  shows  the  axes  for  a  rectangle.  The  short,  diagonal  axes 
are  subsumed  by  the  horizontal  and  vertical  axes,  that  arc  the  global  symmetries. 
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Figure  3.  a.  The  smooth  locus  of  local  symmetries  for  a  portion  of  contour  containing  a 
corner,  b.  The  candidate  axes  for  a  rectangle.  The  small  diagonal  candidate  axes  are  subsumed, 
leaving  the  (global  symmetry)  axes  as  smoothed  local  symmetries,  c.  The  smoothed  local  symmetries 

of  a  square. 

Notice  that  in  case  the  rectangle  were  a  square,  the  diagonal  axes  would  not  be 
subsumed  (Figure  3c). 

Figure  4  shows  some  examples  of  smoothed  local  symmetries.  The  smoothed 
local  symmetries  are  indicated  by  their  axes.  It  should  be  remembered  that  such 
axes  are  merely  graphic  illustrations,  since  they  do  not  show  the  set  of  parameters 
and  symbolic  description  that  are  associated  with  the  smoothed  local  symmetry 
(see  Figure  1  and  Section  2.3). 

Originally,  Brady  [1983,  page  56]  suggested  that  subsumed  smoothed  local 
symmetries  should  be  deleted.  Instead,  we  now  accord  them  lesser  importance  in 
a  hierarchical  representation  of  the  shape.  The  diagonal  axes  in  Figure  3b,  for 
example,  provide  information  about  the  corners  of  the  rectangle,  which,  though  less 
important  than  the  global  symmetries  of  the  shape,  still  convey  useful  information. 
For  example,  Figure  5a  shows  an  acute  angled  corner  discontinuity  on  a  variety 
of  contour  fragments.  The  differences  between  the  contour  fragments  is  explicitly 
represented  by  region  descriptors  such  as  beak  (see  below).  Figure  5b  shows  an 
occluded  instance  of  a  corner.  The  actual  curvature  discontinuity  is  not  present, 
but  it  is  implied  by  the  monotonically  decreasing  width  of  the  region  and  its  small 
absolute  value. 

Smoothed  local  symmetries  were  developed  from  the  ideas  underlying  the  SAT 
and  generalized  cylinders.  Brady  [1983,  pages  53  -  56]  discusses  the  need  to  make 
axis  smoothness  explicit  and  to  refine  the  definition  of  local  symmetry  implicit  in 
the  SAT.  Most  importantly,  all  of  the  local  symmetry  axes  are  made  explicit  for 
smoothed  local  symmetries. 

The  SAT  is  defined  semi- constructively  and  topologically  as  the  union  of  the 
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Figure  4.  Examples  of  smoothed  local  symmetries. 

centers  of  maximal  disks  that  touch  at  least  two  points  on  the  bounding  contour 
of  a  shape.  Figure  6  defines  normal,  branch,  and  end  points  of  the  SAT.  The  SAT 
is  a  piecewise  smooth,  single,  forked  spine.  It  is  not  obvious  a  priori  that  the  SAT 
is  piecewise  smooth;  but  that  has  been  shown  by  Bookstein  [1979].  Branch  points, 
where  the  maximal  circle  touches  the  contour  in  more  than  two  points,  are  a  major 
cause  of  the  problems  of  the  SAT,  giving  particularly  poor  responses  when  there 
is  a  discontinuity  on  the  bounding  contour  of  a  shape.  Figure  7  shows  the  SAT  of 
a  rectangle.  Note  that,  unlike  the  smoothed  local  symmetry  representation  (Figure 
3b),  neither  of  the  global  symmetry  axes  are  found.  Blum  and  Nagel  [1978,  page  169] 
note  that  “the  SAT  is  not  the  simplest  description  for  rectilinear  figures”.  Branch 
points  were  originally  introduced  into  the  definition  of  the  SAT  as  a  discriminant 
for  isolating  sub-shapes  (the  name  comes  from  finding  branches  off  the  trunk  of  a 
tree).  They  do  not  perform  that  task  reliably  either.  Some  other  differences  between 
the  SAT  and  smoothed  local  symmetries  are  presented  in  [Brady  1983]. 

Figure  8  compares  smoothed  local  symmetries  with  the  SAT.  Figure  8a  shows 
a  triangle.  The  SAT  makes  explicit  the  center  of  the  inscribed  circle.  The  smoothed 
local  symmetries  are  the  median  lines.  If  the  triangle  is  isosceles,  a  median  line  is 
a  global  symmetry.  The  SAT  of  an'  ellipse  is  the  major  axis  (Figure  8b),  while  the 
smoothed  local  symmetry  makes  both  axes  explicit.  Figures  8c  and  8d  show  how 
curvature  discontinuities  on  the  contour  of  a  shape  cause  unintuitive  branch  points. 
The  dotted  lines  are  exterior  portions  of  the  SAT  or  smoothed  local  symmetries. 
As  we  shall  see  in  Section  6,  they  are  useful  for  determining  subshape  joins. 


2.3.  Analysis  of  smoothed  local  symmetries 

Suppose  that  the  point  A  in  Figure  2a  is  r/  in  some  vector  frame.  Let  the 
unit  vector  in  the  direction  DA  be  u(s)  and  let  the  length  of  AB  be  h[s),  where  s 
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Figure  7.  The  SAT  of  a  rectangle.  Note  that,  unlike  the  smoothed  local  symmetry 
representation  shown  in  Figure  3b,  the  SAT  fails  to  make  explicit  either  of  the  global  symmetries 
of  the  rectangle.  The  SAT  forks  at  branch  points  that  signal  the  corners  of  the  rectangle. 

indicates  distance  along  the  smoothed  local  symmetry.  The  point  P  at  which  the 
local  symmetry  is  recorded  is  Tp  —  +  i*#).  By  definition  of  the  local  symmetry, 

the  angle  a  between  u  and  the  outward  normal  at  A  is  equal  to  that  between  u 
and  the  inward  normal  at  B.  Aside  from  the  global  coordinate  frame  in  which 
and  Tp  are  expressed,  we  introduce  a  moving  frame  with  origin  P,  collinear  with 
u,  and  measure  angles  counter-clockwise  from  u. 

The  tangent  at  A  is  given  by 


Since  ds^  is  a  unit  speed  parameter  for  the  contour,  is  a  unit  vector. 
Similarly  tfl  is  a  unit  vector.  Suppose  that  a  step  ds  along  the  local  symmetry  axis 
at  P  corresponds  to  a  step  ds ^  at  A  and  a  step  —dsp  at  B.  We  use  a  minus  sign 
to  indicate  a  global  direction  of  traversal  of  the  curve.  Bookstein  [1979]  shows  that 
the  steps  at  A  and  B  are  in  opposite  directions.  We  denote  ^  by  s\,  and  by 
s'B.  Note  that  s\  ■  s'p  is  negative. 

With  respect  to  the  moving  coordinate  frame  centered  on  P,  the  tangent  tp 
at  B  is  at  angle  n/2  —  a  and  is  at  a  —  rr/2.  It  follows  that 


u  \(t-i  ~  *b)  =  O. 


Hence  u  is  perpendicular  to  the  unit  vector 

t=  (ty4  ~ 

2  cos  a 
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We  assume  that  |q|  <  jt/2  for  a  local  symmetry.  In  general,  t  is  not  the  tangent 
t p  to  the  smoothed  local  symmetry  at  P  since 


tp  =  ~Tp 

f’  ■  (2-1) 

Suppose  that  tp  is  at  angle  0(s)  counter-clockwise  from  t,  that  is,  at  angle  <j>  —  \ 
from  u.  Then,  forming  the  dot  product  of  Equation  (2.1)  with  t  gives 


||tp||cos^  =  (s^  +  s'B)- 


By  forming  the  cross  product  of  Eq.  (2.1)  with  t/\  and  t/j,  we  find 
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Figure  9.  The  smoothed  local  symmetry  of  two  concentric  circles  of  radii  R  and  r.  The  ratio 
da^/dag  is  equal  to  R/r,  and  can  be  arbitrarily  large. 


and 


i  __  Jfcp  X  tB|| 
A  IN  X  tB|| 


-  2||tp|| 


sin  a  -f  4 
sin  2a 


(22a) 


j  __  JN  X  Ml 
8  IN  x  Ml 


=  *IM 


sin  a  —  4> 
sin  2a 


(226) 


The  ratio  s'^/s'g  —  ds^/dsg  can  be  arbitrarily  large,  as  can  be  seen  in  Figure  9, 
which  shows  the  smoothed  local  symmetry  of  two  concentric  circles,  the  ratio  of 
whose  radii  is  large. 

It  follows  from  Eqs.  (2.2)  that 


tt\  sin  a  -f  ^ 

tty  sin  a  — 

apart  from  the  special  case  (like  concentric  circles)  when  a  =  ^.  It  follows  that 


tan  d>  =  tan  a 


SA  ~  *B 

•a  +  *'b' 


(2.3) 


H 
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From  rA  —  rp  =  h(s)u,  we  deduce 

/l'(s)  =  (tyl  •  U )s'A  +  (tfl  ■  u)8'jU 

=  sin  a{s'A  +  s'H)  .  (2.4) 

=  2||ty 1 1|  tan  a  cos  <f> 

For  example,  if  <j>  —  0,  as  it  is  for  a  global  or  flexed  symmetry,  then  h'  =  2  tan  a, 
as  expected.  Suppose  that  is  in  direction  <f>A  with  respect  to  some  base  direction, 
and  tj  is  in  direction  <f>u .  Since  <f>A  —  <f>p  —  2a  —  n,  we  find 


2a'  —  4>a~  4>b 
—  kas'a  ~  kbs'b> 


(2.5) 


where,  ka,  respectively  kb,  is  the  curvature  of  the  contour  at  A,  respectively  B. 
Similarly,  the  curvature  of  the  smoothed  local  symmetry  axis,  Kp  is  given  by 


*P  —  4>  +  2^KaSA  +  *b8!b)  (2-6) 

Of  the  parameters  introduced  so  far,  a,  a',  h,  h! ,  ka,  and  Kp  are  directly 
available.  In  general,  dsA  and  dsp  are  small,  so  their  computation  is  poorly 
conditioned  because  of  quantization.  It  seems  that  <f>  is  inevitably  computed 
indirectly  from  points  P  and  the  tangents  at  local  symmetries.  Finally,  Kp  and  <f>' 
involve  second  differential  quantities.  We  solve  the  linear  equations  (2.4)  and  (2.5) 
to  find 


so  by  Eq.  (2.3) 


s\ 4  h'Kp  +  2a' sin  a 

SB  Nka  —  2a'  sin  a  ’ 


tan  <j>  =  tan  a 


h'[KB  —  «a)  +  id  sin  a 

+  *b) 


(2.7) 


(2.8) 


Consider,  as  a  special  case,  a  (non-flexed)  global  symmetry.  The  axis  curvature 
Kp  is  everywhere  zero,  as  are  <f>  and  ^'.  Also,  —  Kp  =  ka  =  k  say.  From  Equations 
(2.5)  and  (2.6) 


d  ~  kA«'a  =  ~kb*'b- 

We  find  that  b'a  —  1/  cos  a,  so  thatj|t/>||  =  1,  hence  h!  —  2  tan  a,  and  d  =  /c/cosa. 
It  follows  that  extrema  of  h  occur  when  a  is  zero  and  they  provide  natural  places 
to  segment  a  symmetric  object  into  its  components.  The  points  where  a  =  0  do 
not  coincide  with  extrema  of  ka  in  general.  Figure  10  illustrates  this.  It  shows  a 
fragment  of  a  symmetric  shape  whose  curvature  consists  of  a  Cornu  spiral  smoothly 
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Figure  10.  Segmenting  a  symmetric  shape.  The  contour  is  the  smooth  join  of  a  Cornu  spiral 
and  a  circle.  The  inflexion  point  coincides  with  the  curvature  maximum  of  the  upper  part  of  the 
contour.  The  minimum  width  occurs  where  a  is  tero. 

joined  at  an  inflexion  to  a  circle.  The  inflexion  points  are  marked.  The  curvature  of 
the  upper  part  of  the  contour  attains  a  maximum  at  the  inflexion  point.  However, 
the  shape  fragment  is  more  naturally  segmented  where  h  is  smallest  (compare 
[Hollerbach  1975]). 

As  a  second  special  case,  consider  a  worm  [Blum  and  Nagel,  1978],  for  which 
h  is  constant  (Figure  11).  It  follows  that  h !  =  h"  =  0,  and  that  a  —  ^  =  <f>’  =  0. 
We  find  that 

KA 
Kp 

*A  * 

Pa  +  pb 

2 

Pb  +  h 

where  p  =  £  denotes  radius  of  curvature. 

Not  surprisingly,  the  description  of  a  worm  amounts  to  the  description  of  a  plane 
curve  that  is  the  axis,  and  a  statement  that  the  cross-section  function  is  constant. 
More  generally,  consider  a  flexed  symmetry  [Blum  and  Nagel,  1978].  It  is  easy  to 
show  that  a  shape  is  a  flexed  symmetry  if  and  only  if  ^  =  0.  The  shape  description 
is  neatly  factored  into  (a)  a  description  of  the  shape  of  the  flexed  symmetry 
axis,  (b)  a  description  of  the  shape  orthogonal  to  the  flexed  symmetry.  That  is, 
in  the  special  case  of  a  flexed  symmetry,  smoothed  local  symmetries  coincide 
with  (two-dimensional)  generalized  cones.  In  general,  however,  the  smoothed  local 
symmetry  representation  accords  more  closely  with  human  perception  than  do 
generalized  cones  (see  for  example'  [Brady  1983,  Figure  10]). 


PP  — 
PA  = 


13 


Indeed,  it  is  clear  that  <f>  is  a  local  measure  of  the  lack  or  symmetry  of  a  shape. 
This  is  because  the  angle  between  t*  and  tp  is  a  —  <j>,  while  the  angle  between 
tfl  and  tp  is  a  +  <f>.  The  difference  between  these  two  angles  is  2 <j>,  and  is  zero 
for  a  flexed  symmetry.  Generalized  cone  parameterizations  are  not  well  suited  to 
describing  asymmetric  shapes. 

Interestingly,  just  the  signs  of  A',  K\,  Kp,  and  a'  allow  us  to  .define  more  than 
the  basic  shapes  proposed  by  Blum  and  Nagel  [1978].  Figure  12  assigns  symbolic 
descriptors  to  primitive  shapes,  defined  by  the  signs  of  the  flanking  curvatures. 
The  beak  primitive  occured  in  Figure  1.  Brady  [1983,  page  62]  compares  the 
descriptors  with  those  proposed  by  Blum  and  Nagel  [1978].  The  primitive  shapes 
have  associated  descriptors,  such  as  a'  and  A',  which  define,  for  example,  the  rate 
of  flaring  of  a  flare  shape.  Heide  [1984]  has  investigated  the  computation  of  these 
symbolic  descriptors. 

2.4.  An  0(n2)  algorithm  for  computing  smoothed  local  symmetries. 

We  have  constructed  one  algorithm  to  investigate  the  competence  of  the 
smoothed  local  symmetry  representation  without  worrying  unduly  about  perfor¬ 
mance.  The  next  section  describes  a  more  practical  algorithm  we  are  developing  in 
a  parallel  effort.  The  time  complexity  of  the  algorithm  described  in  the  remainder 
of  this  section  is  0(n2),  and  consists  simply  of  testing  every  contour  point  against 
every  other  to  find  local  symmetries. 

More  precisely,  consider  fixing  contour  point  A  in  Figure  13,  and  testing 
whether  the  point  B  forms  a  local  symmetry  with  A.  As  in  the  previous  section,  we 
set  up  a  local  coordinate  frame  BA  and  measure  angles  counter  clockwise.  Denote 
the  unit  vector  in  the  direction  BA  by  u(t),  where  t  denotes  arclength  measured  at 
By  and  let  the  angle  between  u  aft'd  the  outward  normal  n/j  be  fi.  B  forms  a  local 
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Figure  12.  Symbolic  descriptors  for  primitive  shapes  defined  by  the  signs  of  the  curvature 
at  the  points  A  and  B  defining  a  local  symmetry.  The  sign  of  h'  defines  the  direction  of  the  shape 
primitive.  The  values  of  a'  and  h'  estimate  the  rate  of  increase  of  the  cross  section  of  the  shape. 

symmetry  with  A  if  and  only  if  n  —  (i  =  a,  which  is  if  and  only  a  +  /?  —  it  =  0. 
Define  the  function  f{t)  =  o(t)  +  (3[t)  —  ir.  More  properly,  /  should  be  denoted 
to  show  that  A  is  fixed.  Finding  local  symmetries  of  A  corresponds  to  finding  tero 
crossings  of  f(t). 

The  simplest  version  of  the  algorithm  considers  every  point  A,  and  tests  every 
point  B  as  a  local  symmetry  with  A.  Figure  14  illustrates  two  problems  that  cause 
us  to  include  in  the  implementation  of  local  symmetry  two  cut-off  tests  at  A  and 
B.  The  low  curvature  of  the  contour  between  A  and  B  in  Figure  14a  gives  rise  to 
the  tiny  candidate  axis  shown.  In  practice,  there  are  many  such  tiny  axes  along 
the  contour,'  and  so  we  suppress  them  by  requiring  the  angle  between  the  normals 
at  A  and  B  to  be  above  a  threshold.  In  Figure  14b  the  curvature  at  point  B  is 
large,  and  so  a  small  portion  of  the  contour  at  B  gives  rise  to  local  symmetries 
along  a  relatively  long  portion  of  the  contour  at  A.  We  currently  solve  this  problem 
by  requiring  the  length  \BA\  to  be  no  greater  than  the  radius  of  curvature  at 
points  A  and  B.  Figure  15  shows  some  examples  of  candidate  axes  produced  by  the 
algorithm. 


Figure  14.  a.  Local  symmetries  in  low  curvature  portions  of  the  contour,  b.  Local  symmetries 
from  a  high  curvature  portion  of  the  contour. 

One  way  to  make  the  0(n2)  algorithm  more  useful  in  practice  is  to  sample 
the  contour  (Figure  16).  This  is  particularly  useful  for  long  portions  of  the  contour 
where  the  curvature  is  small.  It  follows  from  the  analysis  in  the  previous  section  that 
the  curvature  of  the  axis  is  also  small.  By  the  mean  value  theorem,  if  /(<*)  •/(&)  <  0 
there  is  a  zero-crossing  between  t  =  a  and  t  —  b,  which  can  be  found  by  binary 
subdivision.  If  /(c)  •  /(d)  >  0,  we  cannot  conclude  that  there  is  no  zero-crossing 
between  c  and  d.  We  can  approximate  /  between  c  and  d,  say  by  a  function 
/ approx  i  and  sample  more  closely  to  find  a  zero  crossing  of  /  only  if  there  is  a 
zero  crossing  of  f approx ■  We  can  use  a  cubic  approximation  to  /  if  we  can  estimate 
/'(c)  and  /'(d).  Now  /'(t)  =  a'  +  0'.  It  can  be  shown  that  o'  —  =  -Kg.  Also, 

0'  =  Kg  +  [h! /h)  cot  0,  from  which  we  find 


/'  =  nB  +  2-cot0. 
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Figure  15.  Examples  of  candidate  axes  found  by  the  algorithm. 


Figure  16.  Sampling  the  contour  to  increase  the  speed  of  the  0(n*)  algorithm.  The  dotted 
line  indicates  an  approximation  to  the  axis  computed  from  the  sampled  points. 

In  the  next  section  we  discuss  a  more  ambitious  algorithm  that  uses  a  curvilinear 
approximation  to  the  contour. 

An  axis  describes  a  portion  of  the  region  subtended  by  the  shape,  and  does  so 
by  interpreting  the  relationship  between  two  portions  of  the  contour  of  the  shape. 


Let  A  be  a  candidate  axis,  and  denote  by  A/  and  the  left  and  right  portions  of 
contour  that  give  rise  to  the  contour.  Just  as  an  individual  point  can  have  several 
local  symmetries,  so  a  portion  of  the  contour  of  a  shape  can  be  the  left  or  right 
portion  of  several  candidate  axes.  Let  Zfi  be  the  region  subtended  by  A.  Now 
suppose  that  B  is  another  candidate  axis,  and  suppose  that  Ag  overlaps  A/.  We 
say  that  A  subsumes  B  if  Ag  is  contained  in,  but  not  equal  to,  A/  and  if  Zb  is 
contained  in  Z /.  The  mapping  from  axis  to  contour  is  in  fact  a  difTcomorphism 
[Bookstein  1979],  and  so  subsumption  can  be  determined  from  the  contour.  Heide 
[1984,  forthcoming]  has  implemented  the  subsumption  operation,  and  demonstrated 
its  use  on  a  variety  of  shapes.  Figure  17  illustrates  the  performance  of  Heide's 
algorithm  on  an  image  of  a  carving  knife.  Figure  17a  shows  the  axes  of  the  three 
smoothed  local  symmetries  found  by  the  program;  Figure  17b  shows  the  cover  of 
the  blade,  and  Figure  17c  the  cover  of  the  handle. 

3.  Computingsmoothed  local  symmetriesby  appraximatingthe 
contour 


We  are  developing  a  program  that  computes  the  SLS  of  a  shape  by  approximating 
the  contour.  One  reason  for  doing  this  is  that  the  computation  of  smoothed 
local  symmetries  can  be  noisy;  computing  smoothed  local  symmetries  from  an 
approximation  to  the  contour  can  significantly  reduce  the  noise.  The  basic  approach 
is  easily  stated:  choose  a  set  of  points  on  the  contour,  and  then  use  them  to 
construct  a  piecewise  smooth  approximation  to  the  contour  using  the  chosen  points 
as  “knots".  We  then  compute  the  smoothed  local  symmetries  of  the  approximation 
to  the  contour.  We  currently  approximate  the  contour  using  straight  lines  and 
circles.  Figure  18  shows  some  example  smoothed  local  symmetries  found  using  the 
algorithm. 
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Figure  18.  Candidate  smoothed  local  symmetries  found  using  the  approximation  algorithm, 
a.  hammer,  b.  a  pair  of  pliers,  c.  a  carving  knife. 

The  knot  points  should  not  be  chosen  completely  at  random.  Consider  for 
example  a  corner  (Figure  19).  The  knot  should  be  placed  at  the  corner  rather  than 
being  placed  on  one  side  of  it,  in  which  case  a  smooth  approximating  curve  has 
to  include  the  discontinuity  at  the  corner.  More  generally,  the  set  of  knots  should 
mark  significant  changes  in  the  curvature  along  the  contour.  Finding  significant 
curvature  changes  is  a  difficult  problem  exactly  analogous  to  computing  a  rich 
representation  of  intensity  changes  in  one  direction  across  an  image  (“the  primal 
sketch”  of  [Marr  1976]).  In  the  next  section  we  sketch  a  method  for  computing  such 
a  representation  of  curvature  changes.  Asada  and  Brady  [1984]  give  more  details. 

Once  we  have  a  set  of  knot  points,  we  can  compute  a  piecewise  smooth 
approximation  to  the  contour.  Asada  and  Brady  [1984]  observe  that,  at  least 
for  the  class  of  tool  shapes  that  we  have  considered  so  far,  locating  the  knot 
points  accurately  is  more  important  than  the  choice  of  smoothing  functions  to 
approximate  the  contour  between  the  knots.  To  date,  we  have  found  that  fitting 
circles  between  the  knots  is  efficient  and  gives  good  results  (see  [Asada  and  Brady 
1984]  for  examples). 

We  try  to  fit  circles  to  the  points  {(z,-,  yj)|l  <  * <  n}  between  adjacent  knot 
points.  Suppose  the  circle  is  (z  —  zo)2  +  (y  —  yo)2  —  r2  =  0.  We  form  the  square 
error  term 

E{x0,y0,r)  =  £  |(z,  -  z0)2  +  (y<  -  yo)2  -  r2J  , 
and  minimize  E  with  respect  to  its  parameters.  We  have 

r2  —  ~  z<>)2  +  H(V»  ~  Vo)2}, 
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Figure  19.  Knot  points  should  mark  significant  changes  of  curvature  along  the  contour, 
a.  A  corner  fragment  on  a  contour,  b.  Approximation  when  knot  points  (marked  by  crosses)  are 
placed  to  either  side  of  the  perceived  corner,  c.  Approximation  when  the  corner  is  made  explicit. 

and,  after  algebraic  manipulation,  find  that  two  linear  equations  must  be  solved 
for  zo  and  yo: 

xo  E  {xi  -  2)2  +  y0  E  (*<  -  2)(y»-  -p)  =  jE(i.?+ V?X*i  “  *) 

*o  E  (*«•  -  *)(»•■  -  y)  +  vo  E  (v.-  -  v?  =  \  E  (xi  +  VtKvi  -  v) 

where  t  is  the  average  of  the  x,-,  and  y  the  average  y,-.  The  circular  approximation 
can  become  poorly  conditioned  when  the  radius  of  the  circle  is  large  (equivalently, 
the  curvature  is  small),  in  which  case  the  contour  is  better  approximated  by  a 
straight  line.  Denote  the  matrix  of  coefficients  of  xo,yo  by  A: 

»  _  f  E  (ii  - 1)2  £  (i,  -  *)(!/,•  -  V) 

[Ete-iKvi-j)  £(vi-S)2  . 

The  condition  number  of  the  matrix  A  =  (a,y)  is  defined  (see  for  example 
[Strang  1980,  p.  282])  to  be  the  ratio  XmaI/Xmin  of  its  eigenvalues.  Since  the 
eigenvalues  of  A  are  the  roots  of  a  quadratic  in  the  coefficients,  it  is  more  convenient 
to  work  with  the  ratio: 


[Xi  -  X2I 
Xj  +  X2 

\/{a ji  -  022)2  +  *a?2 

T  7M A) 


\i  ’.v*.L  v-  ','V  •  -  *y»i "  :i\-i 
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Figure  20.  Geometry  for  local  symmetry  of  a  point  P  and  a  circular  arc.  The  point  ( xc,yc ) 
is  the  center  of  the  circular  arc,  and  R  is  the  local  symmetry  of  the  point  P  and  the  circular  arc. 

where  the  X,-  are  the  eigenvalues  of  A.  If  L  is  less  than  a  threshold  (currently 
0.96),  corresponding  to  the  condition  number  of  A  being  less  than  5,  we  accept  the 
circular  approximation,  else  we  fit  a  straight  line  to  the  points  (ij,  y«). 

In  the  previous  section  we  described  a  0(n2)  algorithm  for  computing  the 
smoothed  local  symmetries  of  a  shape.  For  each  point  along  the  contour  we  compute 
the  set  of  local  symmetries.  We  can  use  the  contour  approximation  to  speed  the 
computation  by  a  large  factor.  Let  A  be  an  arbitrary  point  on  the  contour.  Instead  of 
comparing  A  against  all  other  points  along  the  contour,  we  compare  A  against  each 
of  the  circular  arcs  that  approximate  the  contour.  Figure  20  shows  the  geometry 
for  determining  the  local  symmetry  of  a  point  and  a  circular  arc.  The  Figure  treats 
the  case  of  positive  curvature  k  of  the  circle.  We  assume  that  the  point  A  is  at  the 
origin  and  that  the  x-axis  is  aligned  with  the  tangent  at  A.  Let  the  center  of  the 
circular  arc  be  (iO)!/o)»  and  let  the  radius  be  r.  Let  the  local  symmetry  between  A 
and  the  point  B  on  the  circular  arc  occur  at  angle  6  measured  counter  clockwise 
from  the  radius  that  is  perpendicular  to  the  y  axis.  It  can  be  shown  that 


0  =  2  tan 
|AP| 


-i  *o  +  r  +  Vo 
*o  -  (r  +  Vo) 


r  ~  (* o  +  Vo) 


2  (r  +  Vo) 


In  these  expressions  it  is  assumed  that  0  <  (APj  <  r,  which  is  the  curvature 
cut-off  condition  referred  to  above.  Similar  analyses  are  possible  in  the  case  k  <  0 
and  for  a  straight  line  segment.  Figure  21  shows  the  result  of  this  on  the  image 
of  a  pair  of  pliers.  The  global  line  of  symmetry  is  quite  visible,  but  is  noisy.  The 
problem  is  that  a  single  point  can  form  local  symmetries  with  several  other  points 
because  of  discretization  and  round-off. 
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Figure  21.  Approximating  the  smoothed  local  symmetries  of  a  pair  of  pliers.  The  local 
symmetries  arc  produced  by  comparing  each  point  on  the  contour  against  the  circular  arcs  forming 
the  approximation.  The  result  should  be  compared  to  Figure  22b,  which  is  the  result  or  computing 
smooth  local  symmetries  from  pairs  of  approximating  area. 

We  can  further  reduce  the  noise  in  the  computation  of  the  smoothed  local 
symmetries,  as  well  as  speeding  their  computation,  by  directly  comparing  two 
circular  arcs  and  computing  an  analytic  expression  for  the  smoothed  local  symmetry. 
This  allows  successive  axes  to  be  chained  together.  Figure  22a  shows  the  geometry 
for  computing  the  analytic  form  of  the  smoothed  local  symmetry  of  two  circular 
arcs.  There  are  various  cases  to  consider,  depending  on  the  signs  of  the  curvature 
of  the  circular  arcs.  In  region  I  (see  Figure  22a)  we  can  show  that  the  locus  of  the 
intersection  of  the  normals  at  local  symmetries  is  an  ellipse.  In  region  II  the  locus  is 
a  hyperbola.  Figure  22b  shows  the  smoothed  local  symmetries  of  the  pliers  (Figure 
21)  computed  by  comparing  arcs  directly. 


4.  Representing  significant  changes  in  curvature 

There  are  two  reasons  for  discovering  and  representing  the  significant  changes 
in  curvature  along  a  contour.  First,  they  provide  a  set  of  knot  points  for  constructing 
a  perceptually  close  approximation  to  the  contour.  Second,  as  we  pointed  out  in 
Section  2.1,  smoothed  local  symmetries  are  both  a  region  based  and  contour  based 
representation  of  shape.  A  contour  based  representation  requires  that  the  curvature 
changes  be  detected,  localized,  and  described  correctly.  This  section  sketches  a 
representation  of  the  significant  curvature  changes  along  a  contour,  a  representation 
that  we  call  the  curvature  primal  sketch.  Further  detail  can  be  found  in  [Asada  and 
Brady  1984]. 

Our  approach  follows  that  of  Marr  [1974].  We  take  ideal  parameterized  versions 
of  a  particular  set  of  curvature  changes  as  our  models  /,•($),  where  /,  is  described 
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Figure  22.  a.  Geometry  for  computing  the  smoothed  local  symmetry  from  two  circular 
area.  b.  The  smoothed  local  symmetries  of  the  pliers  computed  by  directly  comparing  pairs  of 
circular  arcs. 

as  a  function  of  the  orientation  of  the  tangent  to  the  contour.  We  then  determine 
analytic  forms  for  G"  *  /,■(»),  where 


C„(t)  = 


2  a2 


is  a  Gaussian  of  standard  deviation  o.  We  use  Gaussian  second  derivatives 
(approximately  equal  to  the  difference  of  Gaussians  [Marr  and  Hildreth  1980]) 
as  filters  because  of  the  analogy  between  curvature  and  intensity  changes.  The 
algorithm  filters  the  contour  with  a  set  of  Gaussians  of  increasing  standard  deviation 
(scale)  to  construct  a  scale  space  representation  of  the  contour  [Witkin  1983,  Yuille 
and  Poggio  1983].  Using  the  analytic  form  of  G”  *  fi  as  a  guide,  we  construct  a 
program  to  find  instances  of  the  curvature  change  model  along  actual  contours.  We 
have  identified  a  set  of  models,  developed  the  analytic  form  of  their  convolution 
with  the  second  differential  of  a  Gaussian  at  several  scales,  and  implemented  an 
algorithm  that  matches  models  against  the  differentiated  curvature  along  an  actual 
contour. 

Figures  23,  24,  and  25  show  the  set  of  ideal  curvature  changes  that  form 
our  current  set  of  models.  There  is  an  exact  analogy  between  the  problem  of 
representing  the  significant  curvature  changes  along  a  contour  and  the  "primal 
sketch”  representation  advocated  by  (Marr  1982]  for  significant  intensity  changes 
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Figure  23.  a.  The  corner  model  consisting  of  two  circular  fragments  of  curvatures  <Cj  and 
*2  enclosing  an  angle  4>-  b.  The  corner  mode)  in  orientation  space,  relating  the  orientation  of  the 
tangent  to  the  curve  to  distance  along  the  curve,  c.  The  result  of  convolving  the  corner  model 
with  the  second  derivative  of  a  Gaussian. 


Figure  25.  The  remaining  models  in  orientation  space,  a.  The  end  b.  The  dent  or  bump.  c. 
The  smooth  join,  which  is  often  difficult  to  detect  or  localise. 


in  a  single  direction  across  an  image.  For  example,  the  corner  shown  in  Figure  23 
is  analogous  to  a  step  change  in  intensity,  while  the  crank  shown  in  Figure  24  is 
analogous  to  a  thin  bar. 

Consider  the  corner  fragment,' defined  (Figure  23b)  by 


,  ,  n  _/*!»  +  e  if  s  <  0; 

Jeorner[S)^-\K28  +  C  +  6  if  s  >  0. 
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Figure  24.  a.  The  crank  model  consisting  of  three  circular  fragments  of  curvatures  <Ci,  k t 
and  k 3  enclosing  angles  <f>i  and  ^21  the  radius  of  the  crank  being  2a.  b.  The  crank  model  in 
orientation  space  c.  The  result  of  convolving  the  crank  model  with  the  second  derivative  of  a 
Gaussian  where  a/a  >  1. 

It  is  tedious,  though  straightforward,  to  show  that 


>/**o{Gc  *  fcorncr)"[s)  =  ^  «eXP(-~)  +  (*2  “  Kl)exp(-™). 

As  Figure  23c  shows,  this  function  has  a  zero  crossing  near  the  corner  separating 
two  peaks  of  opposite  sign.  The  distance  along  the  s-axis  between  the  zero  crossings 

is 

d 

° corner  —  » 

<P 

The  formula  for  the  height  of  the  side  lobes  is  complex.  In  the  special  case  kj  —  *2, 
it  reduces  to 


corner 


i  1*1 


Figure  26  shows  some  typical  variations  of  the  corner  mode). 
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Figure  26.  Variations  of  the  corner  model. 

As  a  second  example,  consider  the  crank  model,  whose  analytic  form  (see 
Figure  24b)  is 

f/c^s  +  aj  +  c  if  a  <  -a; 

/cranfc(s)  =  «2(«  +  a)  +  C  +  if  -O  <  8  <  OJ 

*3(s  -  a)  +  (2a«2  +  C  +  <f>\  4-  $J2)  if  a  >  a. 

The  crank  is  analogous  to  a  “thin  bar”  intensity  change  in  the  intensity  primal 
sketch.  It  is  even  more  tedious  to  show  that 

y/2na{G0  *  /e,oni)"(s)  =  -4{*l(«  +  a)exP  ~'8 2pi^~ 

f  .  (s-a)2\ 

-f«i2(s-a)exp-^^-J 

.  .  (#  —  a)2 

+  (^3-«2)exp - 

.  .  (s  +  a)2 

+  (*2-*i)exp - ^3— 

The  qualitative  form  of  the  function  ( Ga  *  f crank)" {6 )  depends  on  the  relative  values 
of  its  parameters,  in  particular  the  degree  of  smoothing  a  and  the  contour  length  a 
of  the  step  of  the  crank.  Figure  27  shows  typical  variation  when  =  k2  =  *3  =  0. 
When  the  ratio  a /a  is  less  than  1/2,  the  crank  is  essentially  signalled  by  two 
independent  corner  responses.  However,  when  a  fa  is  greater  than  one,  it  produces 
a  strong  central  peak  with  two  side  peaks  of  opposite  sign  that  are  at  most  half 
the  height  of  the  central  peak. 
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Figure  27.  Second  derivative  of  a  crank  after  Gaussian  smoothing.  The  edges  of  the  crank  are 
assumed  to  be  straight.  The  responses  vary  according  to  the  ratio  a /a,  the  exact  values  being 
shown  to  the  side  of  the  figures. 


Figure  28.  Response  of  a  straight  line  end  as  it  varies  with  o/a.  We  assume  that  each  it* 
in  Figure  23  is  *ero,  and  that  the  4>j  —  w/2.  The  responses  are  for  «r/o  =  0.5,0.75, 1.0, 1.25. 


We  have  similar  analyses  for  the  three  models  shown  in  Figure  25.  A  smooth 
join  generates  a  peak  of  height  (*2  —  *i)/(<J\/27r).  Only  when  the  curvatures  flanking 
the  smooth  join  are  very  different  can  this  be  reliably  found.  This  is  consistent  with 
human  perception  of  such  smooth  joins.  Figure  28  shows  the  second  differential  of 
a  Gaussian  convolved  with  an  end  whose  edges  are  straight.  As  in  Figure  27,  the 
response  varies  with  a /a.  Finally,  Figure  29  shows  the  response  to  a  bump.  If  the 
;«£•■  bump  is  long  enough,  the  response  is  equivalent  to  two  separate  edges  of  opposite 

sign. 
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Figure  30a  shows  the  second  derivative  of  the  smoothed  orientation  at  a  variety 
of  scales  a  along  the  contour  of  a  hammer.  The  model  instances  described  above  are 
marked  on  the  graphs  shown  in  Figure  30b.  In  Figure  30c  we  show  the  knot  points 
that  mark  the  significant  curvature  changes  along  the  contour  of  the  hammer  at 
the  different  scales.  Finally,  in  Figure  30d  we  show  the  smoothed  local  symmetries 
resulting  from  applying  the  approximation  algorithm  described,  in  the  previous 
section  given  the  knot  points  shown  in  Figure  30c. 

5.  Examples 

In  this  section  we  show  some  examples  of  the  smoothed  local  symmetries 
computed  for  a  set  of  hand  tools.  Figure  31  shows  a  screwdriver.  The  global  axis 
of  symmetry  is  found  by  the  algorithm.  Note  that  at  the  coarsest  scale,  the  two 
corners  forming  the  end  of  the  handle,  and  the  (primitive)  end  of  the  blade,  are 
discovered,  as  are  the  two  cranks  defining  the  join  of  the  handle  and  blade.  Figure 
32  shows  a  tack  hammer.  The  head  of  the  hammer,  consisting  of  the  two  striking 
surfaces,  is  easily  found.  So  is  the  handle.  The  join  of  the  handle  and  the  head  is 
a  side-end  join,  and  can  be  found  using  the  method  sketched  in  the  next  section. 
Figure  33  shows  a  carving  knife  (compare  Figure  17)  and  a  cheese  knife. 

6.  Finding  sub-shapes 

The  shapes  that  populate  our  world  have  identifiable  sub-shapes.  A  hammer, 
for  example,  has  a  head  and  a  handle.  The  head  consists  of  a  striking  part,  that  has 
a  flat  surface,  and  a  counterweight  part,  whose  shape  varies  considerably  between 
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Figure  30.  a.  The, 'second  derivative  of  the  orientation  along  the  contour  of  a  hammer, 
smoothed  at  a  variety  of  scales,  corresponding  to  a  =  5,8,11,22.  b.  The  model  instances  are 
shown  in  a  tree  diagram  as  they  arc  tracked  over  the  different  scales,  c.  The  knot  points  that 
correspond  to  the  model  instances  in  (b).  d.  The  smoothed  local  symmetries  resulting  from 
applying  the  approximation  algorithm  described  in  the  previous  section  given  the  knot  points 
shown  in  (c). 


hammers.  Our  current  work  aims  to  make  sub-shapes  explicit.  In  this  section,  we 
report  our  progress.  Figures  34  through  36  show  the  representation  that  we  are 
aiming  at.  In  Figure  34  we  show  how  sub-shape  information  at  a  single  scale, 
indicated  by  the  part-of  links,  is  integrated  with  the  smoothed  local  symmetry 
representation  discussed  in  the  previous  sections.  Figure  35  shows  how  a  shape  can 
be  represented  at  several  scales. 

Figure  36  suppresses  geometric  detail  to  show  other  kinds  of  relationships, 
particularly  part-of  relationships,  a-kind-of  specialization  relationships,  and  con¬ 
straints  between  elements  of  the.  representation.  A-kind-of  links  indicate  class 
constraints.  Instance  links  enable  us  to  index  model  parts  from  smoothed  local 
symmetry  representations  of  shape  fragments.  Other  attributes  of  a  shape  are  under 
investigation.  One  such  is  a  set  of  motor  programs  that  suggests  for  a  model  such 
as  a  hammer  how  it  might  be  u*d  and  for  what  purpose.  Part-of  and  a-kind-of 


Figure  31.  The  smoothed  local  symmetries  found  for  a  screwdriver. 


relationships  have  been  explored  extensively  in  knowledge  representation  studies 
in  Artificial  Intelligence.  In  particular,  ACRONYM  [Brooks  and  Binford  1980]  is 
a  major  contribution  to  the  development  of  representations  of  object  models,  and 
it  has  heavily  influenced  our  work  at  the  level  indicated  in  Figure  36.  All  parts 
in  ACRONYM  models  are  (simple  instances  of)  generalized  cones,  whereas  in  our 
work  they  are  smoothed  local  symmetries,  as  shown  in  Figures  34  and  35.  Following 
ACRONYM,  wc  incorporate  constraints,  in  the  form  of  bounds,  between  links.  For 
example,  there  are  bounds  on  the  relative  sizes  of  the  head  and  handle  of  a  hammer. 
It  would  be  surprising  to  find  a  sledge  hammer  with  a  two-inch  long  handle,  or  to 
find  a  half-inch  wrench  with  a  six. foot  handle.  An  important  part  of  ACRONYM 
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Figure  34.  The  representation  of  the  shape  of  a  hammer.  The  integration  of  part-of  links 
defining  sub-shapes  with  the  smoothed  local  symmetry  representation  of  the  contours  of  and 
regions  subtended  by  the  sub-shapes. 

was  the  program  of  Brooks  [1982]  for  determining  bounds  on  expressions  given 
bounds  on  their  variables. 

We  have  not  yet  investigated  .the  incorporation  of  yet  other  sorts  of  information 
into  our  models.  Such  information  might  make  explicit  surface  markings  or  textures, 
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Figure  35.  The  hammer  is  represented  at  several  scales,  here  two.  At  the  coarsest  scale, 
the  handle  is  simply  represented  as  a  symmetric  shape  with  constant  cross  section.  At  the  finer 
scale,  the  shapes  or  the  handle  and  head  are  refined. 


for  example.  As  we  noted  in  the  Introduction,  in  practical  applications,  the  model 
database  could  be  very  large. 

Our  representation  proposes  four  major  modifications  to  the  ACRONYM 
representation.  First,  we  use  smoothed  local  symmetries  instead  of  generalized 
cones.  The  reasons  for  this  are  developed  in  (Brady  1983].  Second,  we  represent 
a  shape  at  multiple  scales,  for  the  reasons  discussed  in  Section  5  and  in  [Asada 
and  Brady  1984].  Third,  we  incorporate  additional  links,  particularly  instance 
links  and  motor  programs,  into  the  representation.  Finally,  unlike  ACRONYM,  we 
explicitly  represent  sub-shape  connections  (Figure  34).  As  we  discuss  more  fully 
below,  sub-shapes  can  be  joined  together  in  a  variety  of  ways,  in  particular  in  ways 
we  call  side-end  and  end-end  [Marr  1977]. 

In  general,  deciding  that  a  portion  of  a  shape  is  a  sub-shape  requires  the 
mobilization  of  application-specific,  semantic  information.  Here,  we  are  concerned 
with  lower  level  processes,  embodying  weaker  semantics,  that  nevertheless  isolate 
sub-shapes  in  a  usefully  wide  class  of  instances.  Several  methods  have  been  proposed 
to  determine  the  sub-shapes  of  a  shape. 

For  example,  the  original  motivation  for  the  concept  of  branch  point  in  the 
symmetric  axis  transform  was  to  determine  sub-shapes.  The  idea  does  not  work 
well  in  practice.  First,  as  we  noted  in  Section  2.2,  branch  points  often  occur  when 
there  is  an  orientation  discontinuity  on  the  contour,  in  which  case  they  do  not  signal 
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Figure  36.  Other  relationships  of  the  representation.  A-kind-of  links  indicate  specialisations 
such  as  class  inclusion.  Function  links  are  under  investigation. 

sub-shapes.  Second,  branch  points  consist  of  a  single  point  that  is  the  center  of  a 
particular  circle,  and  so  their  placement  is  highly  sensitive  to  noise  on  real  contours. 
In  other  work,  Nevatia  and  Binford  [1977]  located  large  changes  in  the  orientation 
of  the  axes  of  generalized  cylinders  to  determine  sub-shapes.  The  algorithm  would 
fail  to  find  the  join  between  the  handle  and  blade  of  a  screwdriver,  for  example, 
because  the  (symmetry)  axis  is  straight.  Marr  [1977]  proposed  that  object  joins  may 
be  classified  as  side-end  or  end-end,  though  the  notions  of  side  and  end  were  never 
formally  defined.  Assuming  that  the  objects  to  be  joined  were  generalized  cones 
with  straight  axes,  he  developed  a  geometric  analysis  of  a  side-end  join  [Marr  1977, 
Theorem  6  and  Figure  23],  and  proposed  a  formal  version  of  the  familiar  matched 
concavities  heuristic  to  find  side-end  joins.  The  test  he  proposed  required  that  the 
angles  at  the  joins  sum  to  n.  The  approach  was  not  implemented.  It  would  be  as 
sensitive  to  noise  as  the  branch  point  approach  of  the  symmetric  axis  transform. 
Many  authors  have  proposed  that  side-end  joins  can  be  detected  by  finding  a  pair  of 
concavities  in  the  shape,  each  indicated  by  a  negative  minimum  of  curvature.  This 
idea  does  not  work  in  practice.  First,  unless  the  contour  is  smoothed  by  a  suitable 
filter  such  as  the  difference-of-Gaussians,  negative  minima  of  curvature  may  be 
caused  by  noise,  giving  rise  to  false  positives.  Second,  the  absence  of  curvature 
maxima  may  give  rise  to  false  negatives.  Figure  37  shows  that  finding  a  pair  of 
concavities  is  a  sufficient,  but  not  necessary,  condition  for  asserting  a  subpart  of  a 
shape. 

In  order  to  find  the  sub-shapes  of  a  shape,  let  us  ask  first  how  two  shapes  can 
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Figure  37.  A  side-end  join,  one  of  whose  coneavilics  is  not  signalled  by  a  significant  negative 
minimum  of  curvature. 
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Figure  38.  a.  Many  shapes  have  alternative  descriptions.  The  shapes  shown  on  the  left  and 
right  are  typically  interpreted  as  shown;  the  center  shape  is  ambiguous,  b.  A  related  problem 
concerns  ambiguity  about  what  is  joined  to  wbat. 


join.  We  postpone  investigation  of  several  important  questions,  such  as:  how  to 
choose  among  different  possible  “parsings”  of  a  shape  into  sub-shapes  (Figure  38a); 
and  how  to  determine  which  is  the  sub-shape  and  which  the  main  shape  (Figure 
38b).  Figure  39  shows  our  model  of  sub-shape  formation:  two  shapes  may  overlap 
with  optional  smoothing  of  their  contours.  Alternatively,  the  shapes  may  be  close 
and  their  contours  smoothly  joined.  Notice  that  it  is  the  smoothing  of  a  contour 
near  a  real  subpart  join  that  invalidates  naive  expressions  of  the  matched  concavity 
heuristic. 
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Figure  39.  Model  of  (side-end)  join  formation.  The  contours  of  two  overlapping  or  nearby 
shapes  may  be  smoothly  joined. 

Terminations 

In  order  to  assert  subparts,  we  need  to  define  terminations  of  a  shape,  so 
that  we  can  distinguish  side  and  end.  In  our  approach,  a  shape  is  represented 
by  a  (finite)  smoothed  local  symmetry.  By  definition,  a  smoothed  local  symmetry 
axis  is  a  maximal  locus  of  local  symmetries.  It  follows  that  a  smoothed  local 
symmetry  axis  partitions  a  shape  into  two  sides,  which  are  those  portions  of  the 
contour  that  generate  the  axis,  and  two  terminations,  which  are  the  remaining  parts 
of  the  contour.  Switching  attention  between  different  smoothed  local  symmetries 
correspondingly  changes  the  parts  of  a  contour  that  are  perceived  as  side  and 
perceived  as  end.  Sides  are  typically  elongated,  which  is  an  alternative  way  of 
saying  that,  all  other  things  being  equal,  we  prefer  descriptions  based  on  a  shape’s 
longest  axes.  Notice  that  the  description  of  a  join  is  not  a  symmetric  relationship 
between  sub-shapes.  In  Figure  40,  the  “handle”  A  is  on  the  side  of  B  but  B  is  on 
the  end  of  A.  Such  alternative  descriptions  can  be  extremely  useful  in  adequately 
characterizing  a  join. 

Currently,  we  distinguish  four  types  of  termination  (Figure  41).  A  termination 
is  considered  open  if  it  is  the  distal  end  of  a  negative  smoothed  local  symmetry 
corresponding  to  a  concavity.  Typically  the  contour  “turns  away”  at  an  open 
termination.  All  other  terminations  are  closed,  and  are  of  three  types.  First,  in 
a  tapered  termination  (Figure  41a)  the  width  function  h[s )  slowly  tends  to  zero. 
Typically,  the  symmetry  angle  a  is  nearly  constant,  and  a  tapered  termination  gives 
rise  to  a  (single)  curvature  discontinuity  of  type  end.  Second,  for  a  blunt  termination, 
the  width  function  tends  to  zero  but  "quite  abruptly”  (compare  Hollerbach’s  [1975] 
and  Heide’s  [1984]  use  of  symbolic  qualifiers  corresponding  to  ranges  of  values). 
Typically,  there  is  an  abrupt  chartge  in  the  symmetry  angle  or.  A  blunt  termination 
usually  generates  a  curvature  discontinuity  that  is  an  end  at  small  scales  but  a 
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Figure  41.  Three  types  of  closed  termination  of  a  smoothed  local  symmetry,  a.  tapered,  b. 
blunt,  c.  abrupt. 

pair  of  corners  at  larger  scales.  Note  that  both  tapered  and  blunt  terminations  are 
signalled  by  curvature  discontinuities  of  type  end.  Finally,  an  abrupt  termination 
(Figure  41b)  gives  rise  to  a  pair  of  curvature  discontinuities  of  type  corner.  In 
general,  nothing  can  be  asserted  about  h(s)  or  a.  Of  course,  there  is  no  absolute 
distinction  between  a  blunt  and  tapered  end,  or  between  a  blunt  and  tapered 
end.  Positive  smoothed  local  symmetries  ("figure”)  have  two  closed  terminations. 
Negative  ones  (“ground”),  corresponding  to  concavities  in  a  shape,  have  an  open 
distal  termination  and  a  closed  proximal  one. 


Subpart  Joins 
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We  propose  two  rules  that  assert  the  existence  or  a  subpart  join,  prior  to 
determining  whether  the  join  is  of  type  side-end  or  end-end.  As  we  have  noted, 
and  as  we  shall  see  further  below,  this  is  because  the  notions  of  side  and  end  are 
intimately  associated  with  (the  global  property  of)  elongation.  Deciding  that  part 
of  a  shape  is  side  or  end  derives  from  choosing  an  sis  to  organise  one’s  description 
of  a  shape. 

The  first  rule  generalizes  the  familiar  matched  concavity  heuristic: 

Rule  1 

if  there  are  two  corresponding  concavities, 

then  the  cover  of  the  sis  associated  with  the  concavities  is  a  subpart. 

The  crucial  problem  is  how  to  assert  the  existence  of  concavities.  There  are  various 
possibilities,  and  they  rely  on  the  fact  that  our  representation  is  both  con; mi r  and 
region  based.  First,  there  may  be  a  significant  curvature  change  of  type  corner. 
Second,  there  may  be  a  primitive  curvature  change  of  the  crank  type  that  indicate; 
a  (small)  concavity.  This  suffices,  for  example,  to  isolate  the  handle  and  shaft  of  a 
screwdriver. 

Finally,  a  concavity  can  be  asserted  even  if  there  is  no  significant  curvature 
change,  if  the  contour  is  corrupted  by  noise,  or  if  there  is  a  significant  curvature 
change  that  is  overlapped.  In  such  cases  we  rely  on  the  region  component  of 
our  representation.  A  concavity  is  a  “negative”  shape  that  has  a  smoothed  local 
symmetry  axis.  Figure  42a  shows  a  matched  pair  of  concavities  and  the  graph  of 
contour  fragments  and  smoothed  local  symmetry  axes  to  which  it  gives  rise.  We 
assert  the  existence  of  a  subpart,  whose  sis  is  p.  if  (i)  we  find  an  instance  of  the 
sub-graph  shown  in  Figure  42a;  and  (ii)  the  contour  fragments  a  and  d  are  smoothly 
extensible  without  a  significant  change  in  curvature. 

There  is  an  analogous  rule  for  matched  convexities  asserting  the  existence  of  a 
negative  subpart  such  as  an  inlet  (see  Figure  42b).  For  example,  the  axes  X  and  i/  in 
Figure  42a,  corresponding  to  the  concavities,  have  been  labelled  with  a  minus  sign. 
Notice  that  reversing  the  sense  of  figure  and  ground  corresponds  to  interchanging 
minus  and  plus.  Figure  42b  is  interpreted  as  a  subpart  that  is  a  negative  shape, 
that  is,  an  inlet. 

Suppose  that  Rule  1  is  applicable.  The  subpart  it  defines  is  called  a  limb  if  it 
is  elongated  and  has  a  closed  termination.  If  it  is  not  elongated,  it  is  described  as 
an  attached  subpart.  Finally,  if  it  does  not  have  a  closed  termination  it  is  called  a 
tubpart  join.  Figure  43  shows  examples. 

Rule  1  is  sufficient  but  not  necessary  for  asserting  the  existence  of  subparts. 
Figure  44  shows  three  example  shapes  in  each  of  which  there  is  only  one  concavity. 
Figures  44a  and  44b  are  end-end  joins,  but  Figure  44c  is  the  side-end  join  shown 
in  Figure  37.  Figure  45  shows  once  more  the  crucial  role  that  elongation  plays  in 
asserting  the  existence  of  a  subpart.  The  fragment  shown  in  Figure  45a  is  perceived 
as  a  join  in  Figure  45b  but  not  in  Figure  45c. 

Rule  2 
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Figure  42.  a.  A  concavity  is  asserted  if  there  is  an  instance  of  the  sub-graph  of  the 
smoothed  local  symmetry  representation  shown  on  the  right,  b.  Interchanging  plus  and  minus 
enables  us  to  assert  negative  subparts  such  as  “inlets”. 


■ 


W/////M 


Figure  43.  a.  A  limb.  b.  an  attached  subpart.  c.  a  subpart  join.  See  text  for  details. 


if  there  is  a  concavity, 

and  there  is  an  sis,  one  of  whose  generating  contours  Banks  the  concavity, 
such  that  the  sis  is  either  elongated  or  has  a  closed  termination, 

then  the  cover  of  the  sis  is  a  subpart. 

If  the  subpart  has  a  closed  termination,  it  is  described  as  a  limb  if  it  is  elongated 
or  an  attached  subpart  if  it  is  not.  Otherwise  the  (elongated)  subpart  is  called  a 
join.  Figure  44  shows  examples  of  object  joins.  As  in  the  case  of  Rule  1,  there  is 
an  analogous  rule  for  convexities  that  signal  inlet  subparts.  Other  rules  are  under 
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Figure  44.  Matched  concavities  provide  a  sufficient,  but  not  necessary,  for  asserting  the 
existence  of  subparts,  a.  an  end-end  join.  b.  an  end-end  join  generated  by  a  crank,  c.  the  side-end 
join  shown  in  Figure  37. 


Figure  45.  a.  a  shape  fragment,  b.  the  shape  fragment  in  (a)  corresponds  to  a  subpart  join, 
c.  the  shape  fragment  in  (a)  does  not  correspond  to  a  subpart  join. 

investigation. 

Recursive  application  of  the  rules 

When  a  subpart  is  asserted,  the  smoothed  local  symmetries  of  the  two  parts 
are  then  (re-)computed.  (Of  course,  there  are  efficient  ways  to  use  the  existing 
information.)  The  crucial  point  is  that  we  inhibit  local  symmetries  between  points 
that  are  from  different  sub  shapes.  This  removes  many  apparently  spurious  axes  from 
the  initial  investigation  of  a  compound  shape.  The  process  is  applied  recursively. 
This  generates  a  further  hierarchical  shape  organisation  that  is  similar  to  that 


Figure  46.  a.  A  notched  rectangle,  b.  The  smoothed  !oca)  symmetry  axes  for  the  notched 
rectangle,  c.  The  result  of  inhibiting  spines  from  the  rectangle  and  notch  shapes  found  by  rule  1. 

advocated  by  Marr  and  Nishihara  [1978]. 

Consider,  for  example,  Figure  46.  It  shows  the  notched  rectangle,  discussed  by 
many  authors  in  connection  with  the  symmetric  axis  transform.  Figure  46b  shows 
the  smoothed  local  symmetry  axes  computed  for  the  notched  rectangle.  There  are 
several  small  axes  that  seem  to  be  spurious.  There  is  also  a  negative  inlet  (triangle) 
shape  that  can  be  found  using  Rule  1.  Crucially,  every  spurious  axis  derives  from  a 
portion  of  the  contour  of  the  inlet  subpart  and  a  contour  of  the  rectangle.  Inhibiting 
them  leaves  precisely  the  description:  an  inlet  into  a  rectangle. 


7.  Conclusion 

We  have  introduced  smoothed  local  symmetries  as  a  representation  of  two- 
dimensional  shape  that  is  both  contour  and  region  based  and  which  has  local 
support.  We  described  two  implementations,  one  of  which  efficiently  computes  an 
approximation  to  the  smoothed  local  symmetry  representation,  and  relies  upon 
computing  the  curvature  primal  sketch  representation  of  the  significant  curvature 
changes  along  a  contour.  The  previous  section  reported  some  initial  findings  on  the 
isolation  of  subobjects. 
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Currently  wc  are  developing  a  database  of  parametric  models  of  objects  that 
enable  them  to  be  recognized  even  when  they  are  occluded.  We  are  investigating 
the  relationship  between  function  and  form  to  support  reasoning  about  objects. 
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